Method and apparatus for travelled distance measuring by a capsule camera in the gastrointestinal tract

ABSTRACT

A method and system for determining a travelled distance by a capsule camera are disclosed. A current global motion vector for a current image in the image sequence is determined, where the current global motion vector corresponds to movement made by the capsule camera between the current image and a reference image associated with the image sequence. A travelled distance by the capsule camera in the GI tract is determined according to the current global motion vector and prior global motion vectors derived for prior images between an initial image and the current image, where the travelled distance is measured from an initial location associated with the initial image to a current location associated with the current image. A method and system for displaying an image sequence captured by a capsule camera are also disclosed. The travelled distances associated with the image sequence are displayed on a display.

CROSS REFERENCE TO RELATED APPLICATIONS

The present invention is related to U.S. Pat. No. 9,936,151, issued onApr. 3, 2018 and U.S. patent application Ser. No. 15/933,375, filed onMar. 23, 2018. The U.S. Patent and U.S. Patent Application are herebyincorporated by reference in their entireties.

FIELD OF THE INVENTION

The present invention relates to estimating travelled distance by acapsule camera in the gastrointestinal (GI) tract based on imagescaptured by the capsule camera when the capsule camera travelled throughthe GI tract.

BACKGROUND AND RELATED ART

Devices for imaging body cavities or passages in vivo are known in theart and include endoscopes and autonomous encapsulated cameras.Endoscopes are flexible or rigid tubes that pass into the body throughan orifice or surgical opening, typically into the esophagus via themouth or into the colon via the rectum. An image is formed at the distalend using a lens and transmitted to the proximal end, outside the body,either by a lens-relay system or by a coherent fiber-optic bundle. Aconceptually similar instrument might record an image electronically atthe distal end, for example using a CCD or CMOS array, and transfer theimage data as an electrical signal to the proximal end through a cable.Endoscopes allow a physician control over the field of view and arewell-accepted diagnostic tools. However, they do have a number oflimitations, present risks to the patient, are invasive anduncomfortable for the patient, and their cost restricts theirapplication as routine health-screening tools.

Because of the difficulty traversing a convoluted passage, endoscopescannot easily reach the majority of the small intestine and specialtechniques and precautions, that add cost, are required to reach theentirety of the colon. Endoscopic risks include the possible perforationof the bodily organs traversed and complications arising fromanesthesia. Moreover, a trade-off must be made between patient painduring the procedure and the health risks and post-procedural down timeassociated with anesthesia.

An alternative in vivo image sensor that addresses many of theseproblems is the capsule endoscope. A camera is housed in a swallowablecapsule, along with a radio transmitter for transmitting data, primarilycomprising images recorded by the digital camera, to a base-stationreceiver or transceiver and data recorder outside the body. The capsulemay also include a radio receiver for receiving instructions or otherdata from a base-station transmitter. Instead of radio-frequencytransmission, lower-frequency electromagnetic signals may be used. Powermay be supplied inductively from an external inductor to an internalinductor within the capsule or from a battery within the capsule.

An autonomous capsule camera system with on-board data storage wasdisclosed in the U.S. Pat. No. 7,983,458, entitled “In Vivo AutonomousCamera with On-Board Data Storage or Digital Wireless Transmission inRegulatory Approved Band,” granted on Jul. 19, 2011. This patentdescribes a capsule system using on-board storage such as semiconductornonvolatile archival memory to store captured images. After the capsulepasses from the body, it is retrieved. Capsule housing is opened and theimages stored are transferred to a computer workstation for storage andanalysis. For capsule images either received through wirelesstransmission or retrieved from on-board storage, the images will have tobe displayed and examined by diagnostician to identify potentialanomalies.

FIG. 1 illustrates an exemplary capsule system with on-board storage.The capsule system 110 includes illuminating system 12 and a camera thatincludes optical system 14 and image sensor 16. A semiconductornonvolatile archival memory 20 may be provided to allow the images to bestored and later retrieved at a docking station outside the body, afterthe capsule is recovered. System 110 includes battery power supply 24and an output port 26. Capsule system 110 may be propelled through theGI tract by peristalsis.

Illuminating system 12 may be implemented by LEDs. In FIG. 1, the LEDsare located adjacent to the camera's aperture, although otherconfigurations are possible. The light source may also be provided, forexample, behind the aperture. Other light sources, such as laser diodes,may also be used. Alternatively, white light sources or a combination oftwo or more narrow-wavelength-band sources may also be used. White LEDsare available that may include a blue LED or a violet LED, along withphosphorescent materials that are excited by the LED light to emit lightat longer wavelengths. The portion of capsule housing 10 that allowslight to pass through may be made from bio-compatible glass or polymer.

Optical system 14, which may include multiple refractive, diffractive,or reflective lens elements, provides an image of the lumen walls onimage sensor 16. Image sensor 16 may be provided by charged-coupleddevices (CCD) or complementary metal-oxide-semiconductor (CMOS) typedevices that convert the received light intensities into correspondingelectrical signals. Image sensor 16 may have a monochromatic response orinclude a color filter array such that a color image may be captured(e.g. using the RGB or CYM representations). The analog signals fromimage sensor 16 are preferably converted into digital form to allowprocessing in digital form. Such conversion may be accomplished using ananalog-to-digital (A/D) converter, which may be provided inside thesensor (as in the current case), or in another portion inside capsulehousing 10. The A/D unit may be provided between image sensor 16 and therest of the system. LEDs in illuminating system 12 are synchronized withthe operations of image sensor 16. Processing module 22 may be used toprovide processing required for the system such as image processing andvideo compression. The processing module may also provide needed systemcontrol such as to control the LEDs during image capture operation. Theprocessing module may also be responsible for other functions such asmanaging image capture and coordinating image retrieval.

After the capsule camera traveled through the GI tract and exits fromthe body, the capsule camera is retrieved and the images stored in thearchival memory are read out through the output port. The receivedimages are usually transferred to a base station for processing and fora diagnostician to examine. The accuracy as well as efficiency ofdiagnostics is most important. A diagnostician is expected to examineall images and correctly identify all anomalies. Furthermore, it isdesirable to gather location information of the anomalies, which isuseful for possible operations or treatment of the anomalies. Whilevarious location detection devices could be embedded or attached to thecapsule device, it is desirable to develop methods for determining thetravelled distance based on images captured.

BRIEF SUMMARY OF THE INVENTION

A method and system for determining a travelled distance by a capsulecamera are disclosed. According to this method, an image sequence isreceived, where the image sequence is captured by the capsule camerawhen the capsule camera moves through a GI (gastrointestinal) tract.Distance information associated with object distances between thecapsule camera and multiple points in a current image in the imagesequence is received. The current image is normalized according to theobject distances between the capsule camera and the multiple points inthe current image to generate a normalized current image. A currentglobal motion vector for a normalized current image in the imagesequence is determined, where the current global motion vectorcorresponds to movement made by the capsule camera between thenormalized current image and a normalized reference image associatedwith the image sequence. A travelled distance by the capsule camera inthe GI tract is determined according to the current global motion vectorand prior global motion vectors derived for prior images between anormalized initial image and the normalized current image, where thetravelled distance is measured from an initial location associated withthe normalized initial image to a current location associated with thenormalized current image.

The travelled distance can be estimated by accumulating capsulemovements in a longitudinal direction of the GI tract based on thecurrent global motion vector and the prior global motion vectors.Furthermore, the capsule movement associated with a target global motionvector in the longitudinal direction of the GI tract can be determinedby projecting the target global motion vector to the longitudinaldirection. Images of the image sequence may comprise panoramic images,where each panoramic image corresponds to a scene in a field of viewcovering 360-degree of a section of GI tract wall around the capsulecamera.

In one embodiment, a global motion vector is derived for a target imageby dividing the target image into blocks for deriving individual motionvectors for the blocks and the global motion vector is derived from theindividual motion vectors. In another embodiment, the global motionvector is derived for a target image by applying affine motion modelbetween the target image and a target reference image.

The method may further comprise providing information associating thetravelled distance with images from the image sequence. For example, theinformation associating the travelled distance with the images from theimage sequence may comprise a current travelled distance and anidentification of a corresponding image.

The travelled distance may be measured in the forward or backwarddirection. When measured in the forward direction, the current image istemporally after the initial image. When measured in the backwarddirection, the current image is temporally prior to the initial image.

A method and system for displaying an image sequence captured by acapsule camera are also disclosed. According to this method, an imagesequence is received, where the image sequence is captured by thecapsule camera when the capsule camera moves through a GI(gastrointestinal) tract. Travelled distances associated with the imagesequence are presented on a display, where each travelled distancecorresponds to an estimated distance measured from an initial locationassociated with an initial image in the image sequence to a currentlocation associated with a current image in the image sequence. One ormore graphic representations being representative of travelled distancesare displayed on the display according to locations on said one or moregraphic representations.

The above method may further comprise displaying an indicator on atleast one of said one or more graphic representations to indicate acorresponding image associated with a target location pointed by theindicator. In addition, the method may comprise displaying thecorresponding image associated with the target location pointed by theindicator on the display. The above method may further comprisedisplaying the corresponding image associated with the target locationpointed by the indicator on the display.

In the above method, the current travelled distance can be accumulatedfrom global motion vectors associated with the initial image to thecurrent image in the image sequence. The global motion vectors arederived from normalized images of the image sequence and each normalizedimage is generated by normalizing a corresponding image according todistances between the capsule camera and multiple points in thecorresponding image.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an exemplary capsule system with on-board storage,where the capsule system includes illuminating system and a camera thatincludes optical system and image sensor.

FIG. 2 illustrates a simplified example of camera geometry, where theimage sensor is placed at the focal plane behind the lens.

FIG. 3 illustrates a simplified example of travelled distance estimationbased on the global motion vector and the camera optical parameters.

FIG. 4 illustrates a simplified cross sectional view of a four lenssub-systems, where the optical axes are 90° apart in the objectivespace.

FIG. 5A and FIG. 5B illustrate two examples of camera-object distance inthe GI tract environment, where in FIG. 5A, the camera is locatedfurther from the object (a polyp) on the GI wall than the example inFIG. 5B.

FIG. 6A illustrates a cross section of the GI tract for a scenario thatcapsule camera is located in the center of GI tract wall, where the GItract wall is modelled as a perfect round tube.

FIG. 6B illustrates another scenario where the capsule is closer to thebottom side of the GI wall.

FIG. 7A illustrates a uniform motion vector field in the image capturedby the panoramic camera in the scenario of FIG. 6A.

FIG. 7B illustrates a varying motion vector field in the image capturedby the panoramic camera in the scenario of FIG. 6B.

FIG. 8 illustrates an example of normalized motion vectors according tothe distance between respective camera and the GI tract wall.

FIG. 9 illustrates a scenario of varying distance between the capsulecamera and the GI tract wall due to the non-smooth GI tract wall as wellas the non-center location the capsule camera.

FIG. 10 illustrates an example of motion vector field for the scenarioin FIG. 9.

FIG. 11 illustrates an example of normalized motion vectors for themotion vectors in FIG. 10 by taking into account of local imagenormalization to take care of varying object-camera distance.

FIG. 12 illustrates an example of a section of the GI tract with alongitudinal direction in the center of the GI tract indicated by a dashline. FIG. 13A-E illustrate various embodiments of displaying travelleddistance information on a display.

FIG. 14 illustrates an example of displaying the travelled distanceinformation along with the image information, where a graphicrepresentation of the travelled distance and the time code of the imageas indicated by a marker is shown in area on a display.

FIG. 15 illustrates another example of displaying the travelled distanceinformation along with the image information similar to FIG. 14.However, the image as indicated by the marker is also shown in area ondisplay instead of the time code.

FIG. 16 illustrates yet another example of displaying the travelleddistance information along with the image information, where theanatomic parts of the GI tract are also labelled across the graphicrepresentation of the travelled distance.

FIG. 17 illustrates an example of a notebook based implementation, wherethe CPU in the notebook will perform the needed processing and thetravelled distance information and/or the corresponding imageinformation can be displayed on the display (i.e., the notebook screen).

FIG. 18 illustrates an exemplary flowchart for determining a travelleddistance by a capsule camera according to an embodiment of the presentinvention.

FIG. 19 illustrates an exemplary flowchart for displaying an imagesequence captured by a capsule camera according to an embodiment of thepresent invention.

DETAILED DESCRIPTION OF THE INVENTION

It will be readily understood that the components of the presentinvention, as generally described and illustrated in the figures herein,may be arranged and designed in a wide variety of differentconfigurations. Thus, the following more detailed description of theembodiments of the systems and methods of the present invention, asrepresented in the figures, is not intended to limit the scope of theinvention, as claimed, but is merely representative of selectedembodiments of the invention. References throughout this specificationto “one embodiment,” “an embodiment,” or similar language mean that aparticular feature, structure, or characteristic described in connectionwith the embodiment may be included in at least one embodiment of thepresent invention. Thus, appearances of the phrases “in one embodiment”or “in an embodiment” in various places throughout this specificationare not necessarily all referring to the same embodiment.

Furthermore, the described features, structures, or characteristics maybe combined in any suitable manner in one or more embodiments. Oneskilled in the relevant art will recognize, however, that the inventioncan be practiced without one or more of the specific details, or withother methods, components, etc. In other instances, well-knownstructures, or operations are not shown or described in detail to avoidobscuring aspects of the invention. The illustrated embodiments of theinvention will be best understood by reference to the drawings, whereinlike parts are designated by like numerals throughout. The followingdescription is intended only by way of example, and simply illustratescertain selected embodiments of apparatus and methods that areconsistent with the invention as claimed herein.

Endoscopes are normally inserted into the human body through a naturalopening such as the mouth or anus. Therefore, endoscopes are preferredto be small sizes so as to be minimally invasive. As mentioned before,endoscopes can be used for diagnosis of human gastrointestinal (GI)tract. The captured image sequence can be viewed to identify anypossible anomaly. If any anomaly is found, it is of interest to identifythe characteristics of the anomaly as well as its location. Accordingly,the present invention discloses an endoscope incorporating a means forestimating travelled distance by the camera within the GI tract based onthe captured images of the GI tract.

In the past, certain kinds of localization components, such as anaccelerator, gyrator, etc., have been used to trace the ingestibledevice in the GI tract as the ingestible device containing thesecomponents passes through the GI tract. However, the subject usuallydoes not remain stationary after swallowing the ingestible device. Whenthe subject moves, these components cannot reliably differentiate themovement of torso or the device in the GI tract. What a componentrecords is the combination of both movements. In order to developreliable travelled distance estimation, the present invention disclosesan image based approach. In particular, the present invention disclosestravelled distance estimation based on global motion vectors derivedfrom an image sequence captured when the capsule camera travelledthrough the GI tract. By using a global motion estimation method, thetrace of the longitudinal direction is the trace of the device movementrelative to the GI tract. Therefore, it records the GI tractlongitudinal curve much more reliably. Although the bowel within thebody is not completely fixed within the torso, however its uncertaintyis much smaller than the entire torso movement.

Motion estimation is a technique that is widely used in videocompression to estimate movement between images so as to compensate themotion and reduce the differences between images. With the reduceddifferences, the bitrate required to encode the video sequence becomessubstantially reduced. The motions in a scene are usually described by amotion field or motion vectors. The motion may correspond to localmotion associated with a moving object within the scene or global motionassociated with a large area or a whole frame. The global motion isoften caused by camera panning or camera motion. For example, in a videosequence captured by a camera on a moving vehicle facing in a directionperpendicular to the moving direction, motion vectors in the images willbe predominantly uniform and the main motion vector corresponds to aglobal motion vector. If the camera optical parameters are known, theglobal motion vector (i.e., movement between two images) can be modelledby the movement of the camera.

FIG. 2 illustrates a simplified example of camera geometry. In a camerasystem, the image sensor is placed at the focal plane 220 behind thelens 210. The camera can capture a scene within the field of viewextending an angle α. The focal length f is the distance between thelens and the image sensor. The focal length often is fixed forendoscopic applications and is known by design. If the distance Dbetween the camera and an object is known, the dimension of an objectcan be determined from the captured image by measuring the size of theobject image in the image. For example, if an object 230 with height His at distance D from the camera, the object image height H can bederived from the object image height h in the image according to:

$\begin{matrix}{H = {\frac{D}{f}{h.}}} & (1)\end{matrix}$

In the above equation, h is measured from the image, the focal length fis known by design, and the distance D is determined by a selecteddistance measuring means such as structured light technology.Accordingly, if the distance can be determined, the object dimensionscan be derived. The object size in the image can be measured in physicaldimension. However, the image is captured digitally and the sizemeasurement may be more convenient in terms of the number of pixels.Since the physical dimension of image sensor surface and the opticalfootprint are known. Also, the number of pixels is known (e.g. 320×240).Therefore, the object image size in the image can be measured in anumber of pixels and converted physical object image size in the image.

As shown above, the object image size in the image depends on the actualobject size and its distance from the camera. A smaller object at acloser distance may appear to have the same size as a larger object inthe image at a farther distance.

FIG. 3 illustrates a simplified example of travelled distance estimationbased on the global motion vector and the camera optical parameters. Inthis example, object 310 is stationary and the camera is moving. At timet1, the camera is at location 320 and the object 310 is in front of thecamera. The object 310 is projected to the center location 330 ofcaptured frame f1. At time t2, the camera is moved to location 340.Object 310 is projected to image location 330 of captured frame f2. Incaptured frame f2, the object location (330′) corresponding to theobject location (330) in captured frame f1 is indicated. The motionvector between object location 330 and object location 330′ can bederived from captured frame f1 and captured frame f2 using motionestimation techniques. In particular, the motion vector corresponds to aglobal motion vector (MV_(g)) can be derived based on the images, wherethe motion vector is measured in number of pixels.

Since the image sensor size and resolution are also known, the actuallength (l) of the motion vector can be determined according tol=|MV|×pl, where |MV| corresponds to the magnitude of the motion vectorand pl corresponds to distance between pixels. The actual travelleddistance L can be measured according to:

$\begin{matrix}{L = {{\frac{D}{f}l} = {\frac{D}{f}{{MV}} \times {{pl}.}}}} & (2)\end{matrix}$

The concept of motion estimation can be applied to images captured by acapsule camera while traveling through the GI tract of a human subject.However, the motion vectors derived from the images of the GI tractcaptured by a capsule camera are far from the idealistic motion vectorsillustrated in FIG. 3 as described above. The capsule camera maycorrespond to a forward-looking camera located at one end of anelongated capsule device. A global motion vector representing the cameramovement may not be derived easily. The capsule camera may alsocorrespond to a panoramic-view camera, such as the capsule camera systemdescribed in U.S. Pat. No. 7,940,973, issued on May 10, 2011. FIG. 4illustrates a simplified cross sectional view of a four lens sub-systems401-404, where the optical axes are 90° apart in the objective space.Lens sub-system 401 covers a field of view 411, lens sub-system 402covers a field of view 412, lens sub-system 403 covers a field of view413, and lens sub-system 404 covers a field of view 414. The sub-imagescaptured by the multiple cameras can be joined together to form a 360°wide-angle image.

When the capsule camera travels through the GI tract, the capsule cameramay not the positioned in the center of the GI tract. Therefore, thedistances from the camera to the GI wall may not be the same. FIG. 5Aand FIG. 5B illustrate two examples of camera-object distance in the GItract environment. In FIG. 5A, camera 511 is located further from object513 (a polyp) on the GI wall 512 than the example in FIG. 5B. Therefore,the object in the image corresponding to the case of FIG. 5A will appearto be smaller than the object in the image corresponding to the case ofFIG. 5B.

FIG. 6A illustrates a cross section for a scenario that capsule camera611 is located in the center of GI tract wall 612. In FIG. 6A, the GItract wall is modelled as a perfect round tube. Each of the panoramiccameras covers a Field of View (FOV) corresponding to θ. As shown inFIG. 6A, the FOV of neighboring cameras is slightly overlapped. Theoverlapped image areas will be processed to form seamless images. Sincethe capsule is in the center of the GI tract, objects on the GI wallhaving the same size should appear to have the same size in front ofrespective cameras. FIG. 6B illustrates another scenario where thecapsule is closer to the bottom side of the GI wall. In this case, forobjects on the GI wall having the same size, the object in image 4(i.e., the image captured by the camera looking downward) will appearmuch larger than the object in picture 2 (i.e., the image captured bythe camera looking upward). On the other hand, the object in images 1and 3 will appear to be smaller than the object in image 4, but largerthan the object in image 2.

In the case of FIG. 6A, if the capsule device travels in thelongitudinal direction of the GI tract, the image 710 captured by thepanoramic camera will show a uniform motion vector field as illustratedin FIG. 7A. In FIG. 7A, panoramic image 710 consists of 4 sub-images asnumbered with the seamless sub-image boundary indicated by a dash line.Since the panoramic camera in this example covers a 360° view, the rightedge (A′) of panoramic image 710 is wrapped around to the left edge (A).In the case of FIG. 6B, if the capsule device travels in thelongitudinal direction of the GI tract, the images 720 captured by thepanoramic cameras will show a varying motion vector field depending onthe horizontal location of the image as illustrated in FIG. 7B. Forsub-image 4, the camera is very close to the GI tract wall and theobject will look much larger than the image that would be captured bycamera on the opposite side (i.e., associated with sub-image 2).Accordingly, the motion vector in sub-image 4 will be much larger thanthe motion vector in sub-image 2. According to conventional motionestimation techniques, the global motion won't be recognized.

In the scenario illustrated in FIG. 7B, the varying motion vectors arecaused by different distance between a respective camera and the GItract wall. The optical parameters of the cameras are known by design.Therefore, if the distance between a respective camera and the GI tractwall is also known, each motion vector can be used to estimate thetravelled distance by the camera according to equation (2) asillustrated in FIG. 3. The travelled distance by the capsule device canbe derived as the average of individual travelled distances estimatedbased on individual motion vectors. In another embodiment, the motionvectors can be normalized based on the distance between a respectivecamera and the GI tract wall. For example, the motion vectors may benormalized with respect to a nominal distance, such as the averagedistance between a respective camera and the GI tract wall (i.e., thedistance between a respective camera and the GI tract wall in FIG. 6A).In FIG. 7B, while the magnitude of motion vectors varies along thehorizontal direction, these motion vectors should be mapped to the sametravelled distance by the capsule camera. In other words, for two motionvectors, MV1 and MV2 for two different image blocks having distances D1and D2 respectively between the respective camera and the GI tract wall,the following equation holds:

$\begin{matrix}{L = {{\frac{D\; 1}{f}{{{{MV}\; 1}} \cdot {pl}}} = {\frac{D\; 2}{f}{{{{MV}\; 2}} \cdot {{pl}.}}}}} & (3)\end{matrix}$

Therefore, the two motion vectors are related according toD1·|MV1|=D2·|MV2|. If all motion vectors are normalized with respect toa nominal distance D, the normalized motion vectors, MV1′ and MV2′ withrespect to D (i.e., the corresponding motion vector with distance D) arederived as:

$\begin{matrix}{{{{{MV}\; 1^{\prime}}} = {\frac{D\; 1}{\overset{\_}{D}}{{{MV}\; 1}}}},{and}} & (4) \\{{{{MV}\; 2^{\prime}}} = {\frac{D\; 2^{-}}{\overset{\_}{D}}{{{{MV}\; 2}}.}}} & (5)\end{matrix}$

FIG. 8 illustrates an example of normalized motion vectors according tothe distance between respective camera and the GI tract wall. Due tonoise in the image, motion vectors derived from the panoramic image 810may not be accurate. Accordingly, the normalized motion vectors may notbe accurate either. However, the average normalized motion vector (820)will provide a more reliable estimation of the true global motion. InFIG. 8, the normalized motion vectors are normalized with respect to theaverage distance between the cameras and the GI tract walls. While theaverage of normalized motion vectors can be used to estimate the globalmotion vector, other techniques may also be used to estimate the globalmotion vector. For example, the median or the dominant motion vector canbe used to estimate the global motion vector.

In the above discussion, simplified GI tract wall model (i.e., acylindrical tube) is used to illustrate that the varying motion vectorsmay be a result of different distance between the GI tract wall and thecapsule camera. Under a realistic environment, the situation may be muchcomplicated. For example, the GI tract wall is far from the simplifiedcylindrical tube model and the distance between the capsule camera andthe GI tract wall varies from location to location. Furthermore, thelongitudinal axis of the capsule camera may not be aligned with thelongitudinal direction of a section of the GI tract where the capsulecamera is located. In addition, the capsule camera may undergo 3Dmovement such as tilting and rotating, which makes the motion estimationprocess much more complicated.

The longitudinal direction can be derived from the motion vectors. Forexample, the motion vectors can be smoothed and the smoothed motionvectors represent the shape of the trace of the GI tract inside thetorso. The information regarding the longitudinal distance travelled bythe capsule camera may give the doctor a good locational information forsurgery for a lesion found in the images.

FIG. 9 illustrates a scenario of varying distance between the capsulecamera and the GI tract wall due to the non-smooth GI tract wall 912 aswell as the non-center location the capsule camera 910. Distance lines921, 922, 923, 931, 932 and 933 illustrate varying distances due to thenon-smooth GI tract wall 912 as well as the non-center location andtilting of the capsule camera 910. While the GI tract walls arenon-smooth, each cross section of the GI tract wall may be modelled as acircle 920 as shown in FIG. 9. The non-smooth surface of the GI tractwill cause local distortion in the image captured by the capsule camera.For example, a dot having a distance line 922 will appear large in theimage than the dot having a distance line 921 or 923 since the dothaving a distance line 922 is closer to the camera than the other twodots.

The non-smooth surface of the GI tract wall will cause inaccuracy forthe motion estimation process. There are various motion estimationmethods known in the field of video compression. Among them, blockmatching has been a very popular search algorithm, where the blockmatching process searches for a best match block in a reference picture(i.e., a previously processed picture) for a given block in a currentpicture. A motion vector is used to specify the movement between the twocorresponding block. The motion may correspond to a simple translationalmovement that represents a block movement by shifting the blockhorizontally and/or vertically by an offset. The motion may correspondto a more complicated movement that involves two-dimensional blockrotations as well as two-dimensional deformations. The affine model iscapable of describing two-dimensional block rotations as well astwo-dimensional deformations to transform a square (or rectangles) intoa parallelogram. This model can be described as follows:x′=a0+a1*x+a2*y, andy′=b0+b1*x+b2*y.In the above equations, (x, y) corresponds to the pixel coordinates ofthe current image and (x′, y′) corresponds to the pixel coordinates ofthe reference image. For each pixel A(x, y) in the current image, themotion vector between this pixel and its corresponding reference pixelA′(x′, y′) is (a0+a1*x+a2*y, b0+b1*x+b2*y). Therefore, the motion vectorfor each pixel is also location dependent. The affine motion estimationis also well known in the art and the details will not be repeatedherein.

As mentioned above, the non-smooth surface of the GI tract wall willresult in different distances between object locations corresponding topixels in an image area and the camera. The local distance variationwill cause negative impact on the accuracy of motion estimation result.Therefore, according to an embodiment of the present invention, thelocal variations in object-camera distances of pixels within a searcharea for motion estimation and the object-camera distances of pixelswithin a current block are normalized before performing the blockmatching process. For example, for a search area as indicated by an arc930 in the cross section, the object-camera distances for pixels withinthe search area are normalized according to the object-camera distancesof individual pixels. The object-camera distance normalization willenlarge a patch at a longer distance and shrink a patch in a closerdistance. The image normalization based on object-camera distance can beperformed on a sub-block basis by dividing the area into sub-blocks(e.g. 4×4 pixels) and each sub-block is associated with an individualobject-camera distance. If the distance for a given sub-block is smallerthan a nominal distance, the sub-block is shrunk proportionallyaccording to the nominal distance-individual distance ratio. If thedistance for a given sub-block is larger than a nominal distance, thesub-block is enlarged proportionally according to the nominaldistance-individual distance ratio. As is known in image processing,pixel interpolation will be required during image enlargement orshrinking (i.e., scaling). After image normalization based on theobject-camera distance, the motion estimation process should result inmore accurate motion estimation results. The image normalization ismainly focused on the distance variations due to non-smooth surface ofthe GI tract walls. This normalization is used in the stage prior tomotion estimation process so as to improve the accuracy of the motionestimation. Accordingly, the above image normalization to compensate thenon-smooth surface of the GI tract wall is referred as local imagenormalization for object-camera distance variation. In one embodiment,the effect of distance variation due to the off-center location of thecapsule camera will be treated after the motion vectors are derived.FIG. 10 illustrates an example of motion vector field for the scenarioin FIG. 9 by taking into account of local image normalization to takecare of varying object-camera distance. Without the local imagenormalization to compensate the effect of the non-smooth GI tract wall,the motion vectors derived would be less accurate.

In FIG. 10, the motion vectors are mostly pointed toward up-right, whichrepresents that case that the capsule device also rotates around itslongitudinal axis. Furthermore, since the capsule device is off thecenter of the GI tract walls, the motion vectors in sub-image 2 aregenerally smaller than the motion vectors in sub-image 4. Therefore, themotion vector normalization illustrated in FIG. 7 and equations (4) and(5) can be applied to normalize the motion vectors in FIG. 10. Inparticular, the motion vector can be normalized with respect to a targetdistance, such as the average distance between the camera and the GItract walls. FIG. 11 illustrates an example of normalized motion vectorsfor the motion vectors in FIG. 10. The average of normalized vectors(1120) can be used as the global motion vector for the capsule device.

Alternatively, the varying camera-GI tract wall in FIG. 9 could benormalized to a unit radius based on measured distance within the FOV,similar to the scenario shown in FIG. 6A. For example unit radius can beset to 1 cm or 1.25 cm.

One intended usage of the global motion vector is to estimate thetravelled distance by the capsule device inside the GI tract. The GItract is folded within human body. From the point of view of thetravelled distance by the capsule device, the capsule device movement inthe longitudinal direction of the GI tract is of interest. Therefore,the movement in the longitudinal direction of the GI tract should bemeasured. In FIG. 12, a section of the GI tract 1210 is shown and alongitudinal direction in the center of the GI tract is indicated by adash line 1220. An image area 1221 associated with the surface of asection of the GI tract is indicated by a thick line on the innersurface of the GI tract wall. A global motion vector 1222 may be derivedbased on the image area 1221. The image area 1221 may correspond to the360° image of the target section. However, the image area 1221 maycorrespond to the partial image of the target section. The global motionvector may be determined by dividing the image area 1221 into blocks andindividual motion vectors are derived first. A global motion vector isthen derived from the motion vectors. However, the global motion vectormay also be derived directly from the image area. For example, theparameters associated with a global motion vector (e.g. parameters foraffine motion model) may be derived using the image area withoutdividing it into blocks.

Since the capsule device is moved in the GI tract under the action ofperistalsis, it may not move accurately along the longitudinaldirection. For example, the global motion vectors derived for varioussections of the GI tract are shown by the arrows. In order to measurethe travelled distance by the capsule device, the motion vectorcomponent in the longitudinal direction has to be determined. The motionvector component in the longitudinal direction can be determined byprojecting the motion vector to the longitudinal direction. For example,motion vector 1222 is projected to the longitudinal direction to obtainthe travelled distance 1223 associated with the motion vector 1222. Theindividual travelled distances derived from an image sequence for the GItract can be accumulated to obtain the total travelled distance.

As is known in the field, the GI tract peristalsis may cause the capsuledevice to occasionally move backward (i.e., “retrograde”) or oscillate.Therefore, the motion vector may occasionally point backward (e.g.motion vectors 1224 and 1225 in FIG. 12). When the capsule device movesbackward, the associated motion vector contributes a negative value tothe accumulated travelled distance.

The results of estimated travelled distance can be present to a user(e.g. a physician) to evaluate the medical images collected by thecapsule device while travelling through the GI tract. For example, thetravelled distance in the longitudinal direction may be displayed as ahorizontal line, where a reference point may be set as the origin. Thehorizontal distance corresponds to a travelled distance from the origin.For example, line 1310 in FIG. 13A corresponds to an example ofdisplaying accumulated travelled distance, where the origin 1311corresponds to first reference location and the ending point correspondsto a second reference location. The first reference location and thesecond reference location may correspond to particular anatomic parts ofthe GI tract. For example, the first reference location may correspondto the duodenum and the second reference location may correspond to theanus. The image being viewed may be displayed in another display windowon a display device. The location on the longitudinal axis of the GItract for the current image may be indicated on the horizontallongitudinal line 1310 by a marker, such as a downward arrow 1320. Theparticular anatomic part may be determined from the capture imagesaccording to the characteristics of these particular anatomic parts. Thehorizontal axis may be labelled according to the measured travelleddistance. For example, the total length from the origin corresponding tothe duodenum to the ending location corresponding to the anus may be 7.5meters. The horizontal axis may be marked every 10 centimeters. Finer orcoarser markers may also be used. In another embodiment, the globalmotion vector associated with the current image may be display as anarrow 1330 in FIG. 13B, where the arrow may point to forward direct orbackward direction (i.e., retrograde movement). Furthermore, thelocation(s) for the retrograde movement may be indicated on thetravelled distance line 1310. For example, a line segment 1340corresponding to the distance of the retrograde movement may be overlaidon the travelled distance line 1310 as shown in FIG. 13C. Alternatively,the retrograde movement can be more explicitly represented by a backwardtransition 1352 from an original travelled distance line 1350 a toanother original travelled distance line 1350 b as shown in FIG. 13D. InFIG. 13E, a different display style is illustrated, where the transition1352 of FIG. 13D is replaced by a line segment 1354 corresponding to thetravelled distance associated with the retrograde movement. While one ormore horizontal lines are used to represent the travelled distance,other graphic representations may also be used. For example, one or morehorizontal bars may be used. In another example, a numerical number maybe displayed to indicate the travelled distance and up/down arrows maybe used to allow a user to select a desired travelled distance. The usermay use an input device such as a computer mouse or buttons to controlthe selection.

The method may also display images or related information along with thetravelled distance information on a display. FIG. 14 illustrates anexample of displaying the travelled distance information along with theimage information, where a graphic representation of the travelleddistance 1420 and the time code of the image as indicated by a marker1424 is shown in area 1430 on a display 1410. FIG. 15 illustratesanother example of displaying the travelled distance information alongwith the image information, where the image as indicated by the marker1424 is also shown in area 1510 on display 1410. FIG. 16 illustrates yetanother example of displaying the travelled distance information alongwith the image information, where the anatomic parts 1610 of the GItract are labelled across the graphic representation of the travelleddistance. Similar to the case in FIG. 15, a user may pick a point in thetravelled distance line (e.g. using a cursor displayed above thedistance line) and the image corresponding to the location will bedisplayed in area 1510 on display 1410 in FIG. 16.

The method mentioned above can be implemented using various programmabledevices such as micro-controller, central processing unit (CPU), fieldprogrammable gate array (FPGA), digital signal processor (DSP) or anyprogrammable processor. A display can be used for presenting the visualinformation related to travelled distance and/or image information.Furthermore, mobile devices such as tablets or smart phones can be usedto implement the method mentioned above since the mobile devices usuallyhave a display and sufficient computational capability to handle theneeded processing mentioned above. A notebook or a computer can alsoserve as the system to support the method mention above. For example,FIG. 17 illustrates an example of a notebook based implementation, wherethe CPU in the notebook 1710 will perform the needed processing and thetravelled distance information 1730 and/or the corresponding imageinformation 1740 can be displayed on the display 1720 (i.e., thenotebook screen).

FIG. 18 illustrates an exemplary flowchart for determining a travelleddistance by a capsule camera according to an embodiment of the presentinvention. According to this method, an image sequence is received instep 1810, where the image sequence is captured by the capsule camerawhen the capsule camera moves through a GI (gastrointestinal) tract. Acurrent global motion vector for a current image in the image sequenceis determined in step 1820, where the current global motion vectorcorresponds to movement made by the capsule camera between the currentimage and a reference image associated with the image sequence. Atravelled distance by the capsule camera in the GI tract is determinedaccording to the current global motion vector and prior global motionvectors derived for prior images between an initial image and thecurrent image in step 1830, where the travelled distance is measuredfrom an initial location associated with the initial image to a currentlocation associated with the current image.

FIG. 19 illustrates an exemplary flowchart for displaying an imagesequence captured by a capsule camera according to an embodiment of thepresent invention. According to the method, an image sequence isreceived in step 1910, where the image sequence is captured by thecapsule camera when the capsule camera moves through a GI(gastrointestinal) tract. The travelled distances associated with theimage sequence are presented on a display in step 1920, where eachtravelled distance corresponds to an estimated distance measured from aninitial location associated with an initial image in the image sequenceto a current location associated with a current image in the imagesequence. One or more graphic representations being representative oftravelled distances are displayed on the display according to locationson said one or more graphic representations in step 1930.

The above description is presented to enable a person of ordinary skillin the art to practice the present invention as provided in the contextof a particular application and its requirements. Various modificationsto the described embodiments will be apparent to those with skill in theart, and the general principles defined herein may be applied to otherembodiments. Therefore, the present invention is not intended to belimited to the particular embodiments shown and described, but is to beaccorded the widest scope consistent with the principles and novelfeatures herein disclosed. In the above detailed description, variousspecific details are illustrated in order to provide a thoroughunderstanding of the present invention. Nevertheless, it will beunderstood by those skilled in the art that the present invention may bepracticed.

The invention may be embodied in other specific forms without departingfrom its spirit or essential characteristics. The described examples areto be considered in all respects only as illustrative and notrestrictive. The scope of the invention is, therefore, indicated by theappended claims rather than by the foregoing description. All changeswhich come within the meaning and range of equivalency of the claims areto be embraced within their scope.

The invention claimed is:
 1. A method of determining a travelleddistance by a capsule camera, the method comprising: receiving an imagesequence, wherein the image sequence is captured by the capsule camerawhen the capsule camera moves through a GI (gastrointestinal) tract;receiving distance information associated with object distances betweenthe capsule camera and multiple points in a current image in the imagesequence; normalizing the current image according to the objectdistances between the capsule camera and the multiple points in thecurrent image to generate a normalized current image; determining acurrent global motion vector for the normalized current image in theimage sequence, wherein the current global motion vector corresponds tomovement made by the capsule camera between the normalized current imageand a normalized reference image associated with the image sequence; anddetermining a travelled distance by the capsule camera in the GI tractaccording to the current global motion vector and prior global motionvectors derived for prior images between a normalized initial image andthe normalized current image, wherein the travelled distance is measuredfrom an initial location associated with the normalized initial image toa current location associated with the normalized current image.
 2. Themethod of claim 1, wherein the travelled distance is estimated byaccumulating capsule movements in a longitudinal direction of the GItract based on the current global motion vector and the prior globalmotion vectors.
 3. The method of claim 2, wherein a capsule movementassociated with a target global motion vector in the longitudinaldirection of the GI tract is determined by projecting the target globalmotion vector to the longitudinal direction.
 4. The method of claim 2,wherein the longitudinal direction associated with a given image isdetermined from motion vector filed associated with the given image. 5.The method of claim 1, wherein images from the image sequence comprisepanoramic images, wherein each panoramic image corresponds to a scene ina field of view covering 360-degree of a section of GI tract wall aroundthe capsule camera.
 6. The method of claim 1, wherein one global motionvector is derived for a target image by dividing the target image intoblocks for deriving individual motion vectors for the blocks and saidone global motion vector is derived from the individual motion vectors.7. The method of claim 1, wherein one global motion vector is derivedfor a target image by applying affine motion model between the targetimage and a target reference image.
 8. The method of claim 1, furthercomprising providing information associating the travelled distance withimages from the image sequence.
 9. The method of claim 8, wherein theinformation associating the travelled distance with the images from theimage sequence comprises a current travelled distance and anidentification of a corresponding image.
 10. The method of claim 1,wherein the normalized current image is temporally prior to thenormalized initial image and the travelled distance is measured in areverse direction that the capsule camera travelled.
 11. The method ofclaim 1, wherein the normalized current image is temporally after thenormalized initial image and the travelled distance is measured in aforward direction that the capsule camera travelled.
 12. A system fordetermining a travelled distance by a capsule camera, the systemcomprising one or more electronic circuits or processors configured to:receive an image sequence, wherein the image sequence is captured by thecapsule camera when the capsule camera moves through a GI(gastrointestinal) tract; receive distance information associated withobject distances between the capsule camera and multiple points in acurrent image in the image sequence; normalize the current imageaccording to the object distances between the capsule camera and themultiple points in the current image to generate a normalized currentimage; determine a current global motion vector for the normalizedcurrent image in the image sequence, wherein the current global motionvector corresponds to movement made by the capsule camera between thenormalized current image and a normalized reference image associatedwith the image sequence; and determine a travelled distance by thecapsule camera in the GI tract according to the current global motionvector and prior global motion vectors derived for prior images betweena normalized initial image and the normalized current image, wherein thetravelled distance is measured from an initial location associated withthe normalized initial image to a current location associated with thenormalized current image.